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1 . SUMMARY 

? 

1 o 1 Results 

m 

The following ( re suits were achieved in Project 0*1 9 5, Computer 

Assisted/Programmed Instruction on the Law. : 

m 

m 

1.1.1. Four programmed instruction (PI) courses were developed 

in the form of printed texts covering the following 
subjects: The California Penal Code; T he Laws of Arrest, 

Search and Seizure; Evidence; and Dangero us V/eapons 
'Control Laws. These texts are appendeHTas part of this report. 

1.1.2. The self-study PI texts were used as a regular part of 
recruit academy courses for both home s^’udy and as a 
primary activity during supervised study Stall periods in 
class. The self-study materials, which ^.placed formal 
lectures as the primary instructional forr&ht, were found 
to be effective and efficient in reaching trj|iAiing goals, in 
reducing overall study time, in streamlini^ih scheduling of 
the recruit academy, in reducing instructor; work load, and 
in increasing the instructor time available^’, or attention to 

the individual learning problems of studentS^ Two experienced 
officers, each of whom lacked prior formaj^training as 
teachers, were able to present the coursesjon criminal law 
(Penal Code) and on the laws of arrest, search and seizure 
with minimal preparation. Student achievenyent was as high 
as in previous recruit schools. Anecdotal evidence indicates 
that student achievement was actually improved, but no data 
could be obtained to verify this statistically.# • 

An opinion questionnaire, completed by the r'ecruits, showed 
that 97% felt the materials were effective in teaching course 
objectives and 71% preferred PI texts to other* forms of 
instruction which they had in the academy. 8£% expressed a 
desire to have programmed instruction in additional subjects. 

• * 

1.1.3. A limited sample suggests that the self- study Materials 
' are effective in reducing the academic failure :i'ate, 

particularly of recruits with poor academic backgrounds 
and r’-ills. In the past, such recruits have hadiparticular 
difficulty with the course on the Pena], Code. Ir\ the 62nd 
Recruit School, two recruits were dismissed for academic 
failure. They had passing scores of 72% and 78.% on the Penal 
Code final examination, and failed because of grades below 

70% in courses taught by conventional means. \ 

♦ 

A 

* 
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1. 1.4. The PI lessons on the Penal Code were used in a pilot 
in-service training program. Each member of the 450 
man Patrol Division was given a lesson per week and 
allotted fifteen minutes of each regular, daily, 30 minute 
roll call period for individual study. 

1. 1. 4. 1 An opinion poll was devised by the Oakland Police Department 
to evaluate the response of experienced officers to this form of 
in-service training. 91% indicated that they felt their 
knowledge had improved. 78% of the responding officers 
indicated a preference for programmed instruction over the 
lecture format which typified previous training programs. 

58% of the officers voluntarily worked on the materials on 
their own time despite the fact that they were not to be 
tested on the materials or held accountable for them in any 
way. 54% of the officers expressed a desire for more on-duty 
time to study the materials. 77% thought the level of the 
Penal Code course was satisfactory, even though it was 
designed specifically to meet the needs of inexperienced 
officers. It is expected that the course on the laws of arrest, 
search and seizure will be even more suitable for this type 
of in-service training of experienced officers. 

1. 1.4. 2 A small control group of experienced officers was given 

pre-tests and post-tests in conjunction with the PI materials. 
The group as a whole showed a 19% improvement in test scores. 
The ease and economy of providing training by means of 
programmed instruction coupled with the major improvement 
in student acceptance of this program over earlier lineup 
and in-service training programs suggest that the method is 
useful for training experienced officers. 

1.1.5 The PI lessons on the Penal Code were used successfully 
for the training of civilian dispatchers employed by the 
Oakland Police Department. The Commander of Communication 
Section in which the dispatchers are employed found the 
training to be effective and convenient and to be of 
significant help x.o the dispatchers in their work. 

1. 1. 6 The presentation of PI lessons at a computer terminal was 
evaluated and found to be equally effective as the printed 
material. However, the computer was found to be less 
efficient in terms of student time and to be more costly 
than the workbook format. Computer presentation of 
instruction was workable and favorably impres s ed students 
but does not appear to be economical or advantageous. 



1.1.7 A diagnostic test on homicide laws was administered at 

a computer terminal. The computer analyzed the results 
and returned to the student a study prescription keyed to 
the PI materials. The method satisfactorily identified studenc 
needs but offered no advantages over written tests sufficient 
to justify the costs for continued use. 

1.1.8 Analyses of the development and use costs of PI materials 
and of computer pre senta.tion of instruction were made and 
are included in this report. 

1. 1.9 The PI materials on the law have been adopted for continuing 
use by the Oakland Police Department. Presentation of the 
materials and methods developed in this project will be a 
major portion of the program of the third annual Institute 
on Teaching Law to the Police Officer at the University of 
California at Davis in August, 1971. Articles on the results 
of the program are in preparation by the project personnel 
and will be submitted to appropriate journals for publication. 



1.2 Conclusions and Recommendations 

The results of this project support the following conclusions: 



1. 2.T 


Programmed, self-study instructional materials can and 
should be used for recruit training and in-service training 
in the police training environment to achieve higher levels 
of proficiency in less time, to reduce scheduling problems 
of both students and instructors, and to reduce instructor 
preparation time. 


1. 2. 2 


Computer presentation of programmed instruction material 
on legal subjects is more costly and somevhat less efficient 
than printed presentation of the same programmed material. 
These disadvantages appear to outweigh the motivational 
novelty, the automatic record keeping, and the ease of 
updating which are advantages of computer- as sisted 
instruction. 


1.2. 3 


The instructional materials developed under this project are 
ready to use and can be adopted immediately by any law 
enforcement agency in California. These materials permit 
the individualisation of classroom instruction without 
changing teacher- s tudent ratios. The materials permit 
the student to proceed at his own pace, they contain 
intermittent testing which identifies student needs and 
evaluates progress during the learning process, and they 
free the instructor to work with students who have special 
needs, interests, or problems. Detailed instructions and 
suggestions for use of these materials for recruit and 
for in-service training are given in Section 2. 2 of this report. 


1.2.4 


The programmed instruction courses developed under this 
project are appropriate to the training needs of many 
civilian employees of police agencies. Jailers and 
dispatchers are typically most in need of such training. 

Because of the economy of packaged PI materials', and the 
ease of scheduling packaged instruction on an individual 
basis, the materials create a means to provide training which has 
often been denied in the past. 


1. 2. 5 


There exists a need for professionally developed, individualized 
training materials emphasizing active student responses 
in the areas of report writing, criminal investigation, and 
patrol procedures. Such materials would be different in form 
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but similar in educational concepts to the materials 
developed under this program. A grant application for 
the development of such individualized and packaged 
training courses is being prepared by the Oakland Police 
Department for submission to the California Council on 
Criminal Justice. 

1.2.6 The following two letters discussing the results of use of 
the PI texts developed under this project are from the 
Commandeis of the Training Division and the Records and 
Communications Division of the Oakland Police Department. 
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CITY OF OAKLAND 



i'A' .1,^=-.-. _ 




POLICE ADMINISTRATION BUILDING • 455 - 7TH STREET • OAKLAND, CALIFORNIA 94607 
Police Department 



July 12, 1971 



American Analysis Corporation 

420 Market Street 

San Francisco, California 

Attention: Dr. Peter Lenn 

Dear Dr. Lenn: 

We have completed tabulating preliminary returns of the questionnaire 
which was issued to all personnel who participated in the recent Penal 
Code programmed instruction course. The following information, although 
not complete, will perhaps serve as a barometer of the efficacy of this 
type of training for police officers. 

Although 450 personnel participated in the course, only 141 
questionnaires have thus far been returned. This is attrib- 
utable to two factors; (1) no control on questionnaire return 
was implemented via signatures, checkoff lists, etc., in order 
to obtain the most impartial information, and (2) random quest- 
ioning of groups of personnel who participated in the program 
as to the relatively small return indicates a substantial num- 
ber of personnel had not completed the programmed workbooks 
and did not wish to fill out the questionnaire until they had 
done so. This latter factor, in my opinion, is important and 
is discussed more fully below. 

Of the 141 questionnaires received, a high percentage (78%) 
indicated a preference for programmed instruction over the 
lecture format previously employed. Most of the officers who 
responded (77%) indicated that they thought the teaching level 
of the material was satisfactory, oven though the course was 
originally designed for recruit officers. This is significant 
and indicates that performance objectives of the course were 
realistically designed around understanding of on-the-s treet 
situations. More than half (54%) of the respondents indicated 
a preference for more time to complete the workbooks than was 
provided during line-up training sessions. 

A factor which is significant by itself and which reaffirms 
the rationale for not completing the questionnaires is the 
relatively high number of persons (58%) who indicated that they 
devoted additional hours of off-duty study towards completion 
of the programmed workbooks. The 'workbooks were completed by 
officers in recruit school during a total time allocation of 
26 hours and, in retrospect, I feel that it was generally an 
impossible task put upon personnel to complete the workbooks 

- 6-8 



PS line-up training in only nine hours of fifteen-minute 
training periods, which time was fraught with interruptions. 

However, it is particularly impressive to me that Personnel 
voluntarily devoted substantial amounts of their 
hours to complete the workbooks, with no benefit to them 
cept a greater knowledge of the Penal Code. This would seem 
todemonstrate two things; (1) a strong desire on the J? ffl 
of line personnel for in-service training and (2) a reaffirm 
StioS in my mind of the minute-by-minute "challenge to contin- 
ue" this kind of instruction, based on its unique teaching 

method . 

Ninety-one percent (91%) of the personnel who returned quest- 
ionnaires indicated that their knowledge of the Penal Code 
improved as a result of their participation in the programmed 

instruction course. 

'Lastly, it is interesting to note the suggestions for addition- 
al subject material which were made in the questionnane re 
turns especially since a substantial number of persons indicated 
a wish instructional material in Search and Seizure (a course 

which was also designed under your project grant). Othei sub 
jects suggested were Patrol Procedures and Report Writing. 

Tn summary the responses to the questionnaire indicated a high degree 
of acceptance and effectiveness of the program and a desire for addition- 
al subjects by this teaching technique. I hope that you will J- V?cl 
questionnaire results gratifying and you iwy be assured that I find 
them indicative of the quality and versatility of the product which 

you have prepared. 



Sincerely , 



/, 

/- 






/ *' 



"V‘*7 y 



J. McArthur 
Lieutenant of Police 
Training Division 
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CITY OF OAKLAND 




POLICE ADMINISTRATION BUILDING • 455-7TH STREET • OAKLAND, CALIFORNIA 94607 
Police Department 



July 2, 1971 



Mr. Peter D. Lenn, President 
American Analysis Corporation 
420 Market Street 
San Francisco, California 94111 

Dear Mr. Lenn: 

I feel I should convey my impression of the programmed 
instruction on the California Penal Code prepared by 
your firm. When it was first presented to my 
Communications Dispatchers in April of this year, it 
was my first opportunity to observe its effects. 

Few of the employees had prior knowledge of the subject 
but they seem to have gotten more lasting information 
from this course than any other. All comments made 
were favorable and those taking this course preferred 
this method over lecturing. 

Since the dispatchers have been working in the Radio 
Room, they have had fewer problems with the Penal Code 
than in any other area. In my opinion, the programmed 
instruction provided by your firm is both convenient 
and effective. 

Sincerely yours, 




Records & Communications 
Division 



AML: je 
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DESCRIPTION AND DISCUSSION OF PROJECT 



2. 1 Programmed Instruction Materials Development 

2.1.1 Concepts of Programmed Instruction 

The methods of programmed instruction are based 
upon the following concepts: 

1. The curriculum is designed to achieve specific 
educational objectives which are formulated in 
terms of operational competence of the student. 

2. Instructional materials are planned for individual use 
thereby permitting scheduling flexibility and self-paced 
progress by the student. 

3. Immediate re-enforcement of learning by requiring 

its application in question and answer situations, problem 
solving or case study. 

4. Refinement of instruction materials through extensive 
trial on sutdents until the curriculum in fact meets the 
educational objectives. 

2. 1. 2 Instructional Objectives 

The objectives listed below were established to guide the 
current development of instructional materials on the law 
related aspects of the California Penal Code, investigation, 
arrest, evidence, search and seizure and the administration 
of justice. Other training experiences, including outside 
reading, lectures, role playing and supervised field 
assignments will lead to the attainment of additional 
objectives of police training and will contribute to the 
attainment of the following objectives. 

At the conclusion of the course, the recruit will: 

1. be able to identify the roles and describe the operating 
procedures of the governmental bodies involved in law 
enforcement, criminal justice and the maintenance of 
order. 

2. (a) be able to identify the elements of a crime as 

presented in a fact situation and be able to list the 
elements that must be present to establish the 
commission of a crime. 
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(b) be able to classify crimes as felonies and/or 
misdemeanors. 

(c) be able to consult the penal code to identify uncommon 
crimes and their elements or to clarify any point 

of procedure. 

(d) be able to apply the procedural sections of the penal 
code as a basis for police action in case study 
situations . 

3. * be able to decide in case studies when to detain, arrest, 

cite, warn or seek a warrant for arrest, and be able to 
discuss the basis for such decisions. 

4. be able to choose reasonable courses of action and 
appropriate levels of force in case studies or dramatized 
situations, having taken into consideration laws, 
ordinances, policies and procedures . 

5. (a) be able to identify proper and improper searches and 

seizures from case studies. 

(b) be able to identify situations in v/hich search wrr rants 
should be sought from those in which a valid search 
may be made without a warrant. 

6. be able to write reports in conformance with departmental 
instructions. 

7. be able to testify properly in court. 

8. be able to read independently and to apply information in 
departmental training bulletins. 

9. be able to answer questions of law such as might arise in 
explaining his actions and informing citizens of their 
rights and obligations under the law. 

10. be able to distinguish situations in which the patrolman 

has the legal and/or departmental authorization to exercise 
his own discretion. The cultivation of good judgement is 
an overall objective that cannot be attained only by the 
materials being developed in this project. Other training 
approaches including lectures, reading, role playing 
and field training will be used concurrently and will be 
directed at development of good judgement. 



i 
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On the basis of these general objectives, specific 
objectives were formulated for each portion of the course. 

These objectives are described in each of the four courses, 
so that the students and their instructors can determine 
what is to be learned and whether it has in fact been 
learned. 

In the case of the course on the Laws of Arrest, Search 
and Seizure and the course on Evidence, the specific 
objectives were originally formulated as a set of test 
questions covering all the skills to be taught. These 
"Criterion Tests" appear as the last section of the text 
on the Laws of Arrest, Search and Seizure and in the form 
of "Progress Check" tests at the 6nd of each of the four 
lessons of the Evidence text. For the Penal Code and 
Dangerous Weapons courses, the objectives were originally 
in narrative form and the criterion tests were developed 
subsequently. The narrative objectives used to design the 
Dangerous Weapons course are included below to illustrate 
the form of the narrative objectives and to permit comparison 
between these objectives and the Criterion Test at the end of that 
course. 
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OBJECTIVES - Dangerous Weapon Control Laws 



At the completion of this lesson the student will be able to 
perform the following tasks: 



1, Determine whether a potentially dangerous implement is a 
dangerous weapon according to the California Penal Code. 

2. Classify dangerous weapons according to type. 



3. Identify’ those weapons, the mere possession of which is 
uni awful . 



S. 12020 

S. 12420 
S. 12020 
S. 12220 



a. blackjacks, slungshot, billy, sandclub, 
sandbag, metal knuckles 

b. teargas weapons 

c. sawed-off rifles, sawed-off shotguns 

d. machine guns 



4. Identify those weapons which are unlawful to be concealed on 
the person or in a vehicle. 



S. 12025 a. handguns, concealable firearms (without a license) 
S. 2006 F&G b, rifles or shotguns with cartridge in chamber 
S. 653k c. switchblade over 2" in blade length 



5. S. 12031 Identify those special situations in which it is a 
misdemeanor to carry loaded firearm. 

6. S. 171c Identify those special situations in which it is a felony 
to carry loaded firearms (Mulford x4.ct) 



7. Identify the classes of persons whose privilege to bear firearms 
is specially restricted. 



S. 12021 

ii 

ii 

S. 12021.5 



a. any person who is not a citizen 

b. any person who has been convicted of a felony 

c. any person who is addicted to the use of a dangerous 

drug 

d. minors lacking parental consent 



8. Identify unlawful explosive devices as defined by the Penal Code 



9. S. 12022 Recognize the occurrence of a separate punishable offense 
during the commission of a felony while armed with a concealable 
weapon. 
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10a. S. 12027 Consult the Penal Code to identify the following 
exceptions to weapon control laws. 

a. clubs and organizations who use target ranges, 
transportation to and from. 

b. clubs, which collect and display guns, antiques, etc. 

c. clubs, civil and military organizations, while parading 
transportation to and from. 

11. Identify crimes involving tampering with serial marks 
on firearms. 

S. 12090 a. tampering with serial number 

S. 12093 b. falsifying identification number 

S. 12094 c. purchase or sale of weapon lacking 

identifying number 

12. Determine whether violations of weapons control laws are 
misdemeanors or felonies. 
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2 . 1. 3 



Developmental Testing 



All of the instructional materials prepared were subjected 
to repeated cycles of trial usage by typical students 
followed by revision. This testing process is accomplished 
by having the student work through a course without help 
while being observed by an educational technologist. The 
observer notes any incorrect responses and the student's 
reaction upon comparing his response to the correct response. 

To observe a student working on PI materials is a fascinating 
experience for an educator. One immediately observes 
that the student is motivated by his success in making 
correct responses. When a student makes a wrong response, 
whether it was due to his own carelessness or to a defect 
in the materials, he loses enthusiasm and concentration. 

In order to obtain an accurate evaluation of the materials, 
the observer must avoid tutoring the student on the material 
before it has been presented in the PI text. On the other 
hand, it is necessary to assist the student over weak points 
in the materials. In this way the entire course can be tested 
without interference by correctable deficiencies. 

The function of testing of programmed instruction materials 
is to determine their educational shortcomings. Each lesson 
is intended to enable the student to acquire demonstratable 
behavior which he was unable to exhibit before taking the 
lesson. Behavioral goals can range from an ability to spell 
"corpus delicti" to being able to determine from a case history 
what crime, if any, has occurred. Testing with typical 
students provides an indication of which training goals have 
been met, which were simply not met, and where the student 
has been trained to behave in an incorrector undesired manner. 
Initial testing with one or two students is sufficient to determine 
major shortcomings. Later, after several cycles of testing and 
revision, it is necessary to test the materials on a larger 
number of students to "validate" the fact that the materials do 
enable the entire target population to reach the training 
objectives. 

Testing of the programmed instruction materials also provides 
the data necessary to develop efficient lessons. By determining 
those places in the lessons where students become confused or 
respond incorrectly, it is possible to repair these shortcomings, 
so that later students will not encounter difficulty and will 
therefore be able to learn more quickly. It is an important 
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concept of education that the learning process not be 
considered a test. One may test before the lessons 
begin to select students and after the lessons have 
ended to determine whether the students can do those 
tasks for which they have been trained. But, during 
the lessons, a student's learning is facilitated by active 
participation in the lessons, by continuous success and 
by lack of confusion. 

Programmed instruction lessons are designed to take 
advantage of these concepts of promoting efficient learning. 
The lessons require the student to respond actively at each 
step of the way. It is the responding that is important to 
learning, not the fact that there is a possibility that the 
student may make an incorrect response. Therefore, it is 
a goal in the development of programmed instruction 
materials that all the students make correct responses to 
all the questions in each lesson. Though every student 
will respond incorrectly occasionally, and some students 
may make a large number of incorrect responses, the 
testing process enables us to develop PI lessons that 
minimize student confusion and provide him with the 
motivation that comes from succeeding in his task of 
learning. A student learns faster if he can be led to making 
the right response to every problem presented to him. 

Unlike conventional texts and lectures, the burden of 
success or failure is assumed to be the responsibility 
of the programmed materials. During the testing and 
revision process, what appears to be student failure 
identifies flaws in the materials under development. The 
result is an instructional package in which student 
success in terms of predefined educational objectives is 
predictable for target student populations. 

1.4 Validation Testing 

The purpose of validation testing is to determine 
whether the PI materials are actually succes sful in 
enabling all of the students to achieve the educational 
objectives. Of the four courses developed, the Penal 
Code text was most extensively tested and the course 
on Laws of Arrest, Search and Seizure, was used with one 
full recruit class. Though the Evidence and Dangerous 
Weapons texts were subjected to trial usage during 
development, neither has received sufficient use to be 
considered validated. However, these materials will be 
used in forthcoming recruit academies and it is expected 
that the results achieved will validate those courses. 
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2. 1.4. 1 First Validation Text of Penal Code Materials 

In this first Validation Test of the Penal Code text, the 
students were given 8 days to cover the material. During 
this time they had five 3-hour study hall periods in which 
they could work or not, as they chose. In the study hall 
periods, they had access to a proctor who was able to 
answer questions and discuss the subject matter knowledgeably. 
Most students spent the study hall hours rather at leisure and 
did the bulk of their studying at home. After the first day, 
student questions were concentrated in the first hour of each 
day's study hall period. This leads to the conclusion that 
the study hall periods could be shortened to a daily 1-hour 
discussion period, during which the problems which the 
students encounter in their home study could be covered. 

On the last of the 8 days allowed for studying the PI materials, 
the students had a 3-hour lecture/discussion session with 
the Penal Code instructor. The following day they were 
given the same final examination on the Penal Code that 
had been used in the recruit academy of the previous summer 
(but omitting the last four questions on the laws of arrest). 

A copy of that examination is included in the Appendix of this 
report. A comparison of the results obtained in the previous 
administration of this test to students who had been given 
traditional instruction with those obtained by the students who 
used the PI materials is shown in Table 1. In addition to 
the numerical results shown in the table, there are numerous 
points of comparison and evaluation that were made and are 
discussed below. 
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TABLE 1 
PENAL CODE 
TEST SCORE DATA 

Note that a different test was used in the 62nd OPD Academy. 

The tests are included in the Appendix of this report. 

Revised 

Conventional Programmed Programmed 

Instruction Instruction Instruction 



OPD Academy 


59 th 


6lst 


62nd 


Number of students 


20 


28 


26 


Elapsed days of course 


13 


8 


6 


Lecture hours 


25 


3 


3 


Number of questions 








rejected 


8 


16 


0 


Highest score 


91 


90 


97 


Lowest score > 


60 


55 


72 


Class average 


78.8 


78.4 


87 



i 



3 

| 



1 

s 



•J 



•t 



1 

I 



;! 
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The students using the PI materials achieved esentially the 
same class average as the earlier group and did so with 3 
hours of lecture instead of 25, and approximately the same 
20 hours of studying. There are only estimates of the 
number of student hours previously spent in studying for the 
Penal Code course, but anecdotal evidence indicates that 
no more time was spent with the PI materials even including 
the study hall hours which were poorly utilized. 

Though the class average was the same, there was a slight 
increase in the spread of the grades in the group using the PI 
materials. Also, there were several more students having 
grades below the "passing" grade of 70. It is believed that 
these differences are due to some' omis sions in the topics 
covered by the PI materials and to certain problems with 
the test. Both of these issues are described more fully 
below. 

The test shown in Appendix 1 has certain peculiarities of style 
that are unusual and require some getting used to. For example, 
true- false questions are presented in pairs and the student must 
first answer both questions and then combine the two answers to 
match one of the four pos sible multiple-choice options presented. 
27 of the 49 questions are of this type. See questions 5 and 6 
for examples. It would seem that when first confronted with 
this unusual method of halving the weight of each true- false 
item and of putting on a one for one guessing penalty, the 
student would be confused and do less well. Prior to the 
earlier administration of this particular examination, the 
students had been warmed up for the instructor's style by 
25 hours of lecture and 3 mid-course quizzes prepared by him. 

A further difficulty in analysing the results arises from the 
instructor's policy of disregarding those questions which are 
missed by more than 50% of the class. Based on an intention 
to be fair, this policy would be even better if it were used as 
an indicator of the shortcomings of either the instruction or 
the test. However, no record can be found of which 8 
questions were disregarded in the earlier use of this test. 

16 questions were missed by more than 50% of the students 
in the present trail and were disregarded in computing the 
class average. These were items: 

2, 14, 15, 20, 21, 24, 25, 29, 31, 32, 37, 40, 42, 43, 45, 
and 49. Of these, ten covered topics which were intentionally 
or unintentionally omitted from the PI materials. 
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(The unintentional omissions are being corrected.) 
The six questions remaining are 15, 25, 31, 32, 40, 
and 42. Except for #15, these are difficult pairs of 
true-false questions. 

The second half of item 25, for example, requires the 
student to notice that: 



"Paying a person who provides a woman for the purpose 
of engaging in sexual intercourse establishes the crime 
of pimping." 

is a false statement because "receiving" rather than "paying" 
is needed to establish the action of pimping. The question 
could cover the same concept more clearly if the student were 
asked whether the payer or the receiver was the pimp. 



It is difficult to establish a correct assignment of the cause 
of these rejections to either the PI materials or the test. 
However, the questions did indicate that strengthening the PI 
coverage of attempts and proximate cause would be helpful. 
This is being done. 



While the results in Table 1 indicate success of the PI materials 
in helping students achieve the same class average in less time, 
this does not give a full picture of the real effectiveness of 
the PI materials because the PI materials and the test have 
different emphasis. Before having available the tool of 
educational objectives, teachers typically make up tests 
which penalize insufficient effort on home assignments and 
insufficient attention in class. The alternative approach, 
which is generally considered a better measure of the 
effectiveness of an educational activity, is to test whether 
the student has achieved the ability to do those things which 
he was to be taught to do. With the former emphasis, it is 
considered reasonable to hold the students responsible 
for any major item of information mentioned in a lecture. 

The latter approach would not allow such a test question 
unless "remembering in lecture" or "remembering that point" 
had been an objective of the course. A specific example in 
the present case is the following. The PI materials were to 
train the student to: 



1. charge suspects in homicide cases with homicide and 

not to make a distinction between murder and manslaughter, 
nor the degrees of those crimes at the time of arrest. 











nor 



2. investigate for the types of facts that would relate 

to establishing or disproving the elements of murder, 
manslaughter, and their various degrees. 

In view of this objective, questions 16 , 17, and perhaps 30, 
which ask the student to choose between the types of homicide 
and their degrees, test a capability which was not an 
objective and was not specifically taught by the PI materials* 

The content of those questions was however used for examples 
in the earlier course, so the students had a better chance to 
know the right answer even without an increased ability to - 
make such distinctions in general. 

Another problem with the test as a full measure of the PI 
materials was the lack of emphasis on those capabilities 
which were emphasized in the PI materials and in which the 
students might be expected to do better than those who had 
not had the PI materials* This applies to such topics as 
deciding which facts in a case establish particular elements 
of a crime, establishing the capacity to commit a crime, and 
establishing the mental element in crimes where criminal intent 
may not be inferred just from the commission of a prohibited 
act. Though the test called for the identification of crimes, a 
process which seems to include matching facts to elements as a 
necessary step toward a correct answer, this is really not an 
adequate test of the desired behavior. The evidence of this 
is that a large number of arrests for properly identified crimes 
do not lead to prosecution because the peace officer did not 
do an adequate job of identifying and then describing which facts 
he used to establish each element of the corpus delicti. It is 
# hoped, but not yet proved, that the PI materials have provided 
the students with a better capability in these areas than was 
previously the case. 

The PI materials as used in this trial did not cover riots, 
disturbing the peace and conspiracies, but these topics were 
involved in several of the test items. These shortcomings 
of the PI test have been remedied. Both the lack of 
familiarity with the instructor's testing methods and the 
fact that some topics were simply not covered in the existing 
version of the PI materials would account for the increased 
spread in the students' grades. This- is because variability 
in the students' initial knowledge and general test-taking 
abilities would have a greater effect in this situation than will 
be the case when the coverage deficiencies of the PI materials 
are corrected and when a test more closely atuned to the 
defined behavioral objectives is used. 
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2. 1.4.2 Second Validation Test of Penal Code Materials 

The PI materials on the Penal Code were expanded 
following the first usage to cover those areas previously 
omitted but considered necessary by the subject matter 
expert who reviewed the results of the first trial. In 
the second validation test, the students again were given 
three hours of lecture and discussion. Twelve hours of 
supervised study hall time was provided in class and the 
students completed their studying as a home assignment. 

The elapsed time of the course was shortened to six days, 
so that it could be completed before the course on the laws of 
arrest was begun. 

This schedule permitted only one week-end for slower 
students to catch-up and complete the lessons. However, the 
final examination on the Penal Code course was not held 
until a week later to allow additional study time, particularly 
for memorizing the elements of the crimes listed in the 
appendix of the course. During the intervening week, the 
students' time was occupied with the PI materials on the 
Laws of Arrest, Search and Seizure and with the beginning 
of the course on report writing. 

The final examination on the Penal Code administered 
to this 62nd Recruit Academy is included in the Appendix 
of this report. A comparison of this test with that used 
in both the 59th (Control Group) and 6lst (First Validation) 
indicates that the testis more intelligible, less tricky, and 
attempts to cover the same material with greater relevance 
to job performance. 

It should be noted that none of the examinations contained 
in the Appendix to this report should be considered ideal 
examinations. Validation of the materials required use 
of existing te st questions . It is strongly recommended that 
local agencies draft their own examinations by modifying 
the criterion tests contained in the PI texts to suit their 
own individual needs. 

Scores on the final examination given to the 62nd Recruit 
School may be considered a better indication of the 
effectiveness of the PI materials than the earlier test 
described in Section 2. 1.4. 1 of this report. The grades of 
the recruits in the 62nd Academy are also shown in Table 1. 
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2. 1.4.3 Effectiveness in Meeting Minimum Objectives 

Among other requirements of the Oakland Police 
Department Recruit Academy, all recruit officers 
must achieve a passing score of 70% or better on the 
Penal Code final and for the recruit school as a whole. 

To date, no recruit has failed an examination on 
material offered through programmed instruction. 

Traditionally, the Penal Code course has been the most 
difficult course offered in the school. Two recruits in 
the 62nd Academy were terminated because of academic 
failure. However, the men passed the Penal Code final 
examination with scores of 72% and 78%, and failed as a 
result of performance in other aspects of the school. This 
limited evidence, observations of instructors in the academy, 
and the range of scores in the validation tests indicate 
that the PI materials effectively reduce the rate of 
academic failure. While the materials, supplemented by 
the informal assistance of an experienced officer,, were 
found to satisfy the needs of all students, they also tended 
to minimize the difference in achievement between the 
most successful and least successful students. The 
orderly presentation of information in the programmed 
instruction texts tends to lower those barriers to learning 
which block students disadvantaged by poor academic 
backgrounds or a lack of refined study skills from 
achieving the minimum academic performance required 
by the Recruit School. The programmed instruction texts 
also identify and faciliatate individualized attention to 
specific learning problems of particular students before 
the moment of the final examination. 

2. 1.4.4 Validation Testing of the Laws of Arrest, Search and 
Seizure Course 

After developmental testing, the course on the Laws of 
Arrest, Search and Seizure was used with the entire 62nd 
Recruit School class. The twenty- six hours of class time 
available for the course were devoted to ten hours of lecture 
and discussion, ten hours of supervised study hall, and 
six hours of role playing. Two examinations were given and 
are included in the Appendix of this report. The "quiz" 
covering the Laws of Arrest, Search and Seizure resulted 
in a class average of 89%. No official final examination is 
given in this course, but questions on the course were 
included in a general mid-term examination. Of the twelve 
questions pertaining to the course included on this mid-term, 
no student missed more than two. The class average was 93% 
on those 12 questions. These results indicate that the materials 
were successful with this group of students. 
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2. 2 Programmed Instruction Materials Applications 

2. 2. 1 Recruit Training Application 

During the validation testing, most the time previously 
scheduled for each course was retained in the Recruit 
School schedule. This provided for the possibility that 
failure of the PI materials might necessitate a return to 
the conventional instructional approach in order to bring 
the recruits to the required level of proficiency in each 
course. As that did not become necessary, the bulk of 
class time was devoted to study hall sessions, in which 
the students worked informally and individually on the 
PI lessons. A police officer or instructor was present 
throughout these periods to answer questions and to 
discuss particular points with individual students. 

This study hall arrangement was found to provide a number 
of important advantages, though the amount of time 
allotted was excessive and could be reduced in the 
future. Providing the students an opportunity to have 
directed, yet informal, discussions with a working 
officer enhanced the relevance of the course and provided 
teaching at an operational rather than academic level. 
Further, because the teacher must only relate police 
actions to the law, he has no need to prepare an 
academically- styled course. It was suggested that the 
instructors assume the attitudes about legal questions 
which they would ordinarily take when performing a police 
function. They were to answer questions, if possible, in 
\ terms of what they would do if faced with an actual incident. 

They were to research or seek legal advice on any 
questionable points and report back to the student. In this 
\ way, the artificiality of classroom training was reduced. 

Recruits were not expected to solve problems in the 
<■„ classroom which an experienced officer could not and need 

not solve on the street. Recruits also obtained practice 
in choosing a course of action in the absence of complete 
information. 

The study hall environment enables students to talk 
in small groups with instructors and with each other. 
Students may pursue those aspects of the course which 
specially interest them and get help on any concepts 
that are bothering them. Since the PI materials call for 
active responses from the student, his inability to respond 
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correctly to just a few frames in succession signals 
that he has taken a wrong mental turn or missed a key- 
concept. In the individualized format made possible by 
the PI, the student can stop when he needs help and get 
it. He can do so without disrupting the progress of others 
or the performance of a lecturer. Lecturers often feel 
that it is their role to present material to the whole class. 
Thus, in a conventional lecture course, the confused 
student is viewed as a nuisance for whom the lecturer does 
not have time to stop. Even the most skillful lecturer is 
frequently faced with the problem of ignoring individual 
problems or wasting the time of many students to help a few. 

A most noticeable difference exists between individualized 
in truction in a study hall setting and a conventional 
lecture. On the one hand, lecturers often complain they 
do not have time to do justice to the subject, let along 
answer the questions of students who have particular 
problems. On the other hand, one finds a signle instructor 
able to provide individualized attention to 30 or more 
students. This neglected advantage of the one room 
schoolhouse allows the instructor to ensure that a student 
has fully grasped a point before the s tudent proceeds to 
the next topic. 

Lectures are particularly inefficient for presenting 
course content. A conventional text is a better way to 
present information. In the ideal learning situation, 
a skilled tutor engages in a continual dialogue with the 
student. PI materials are an attempt to package the 
advantages of a tutor at the cost of a text. 

The distinct and important role of the lecture in training 
is to provide the students with a performance by a person 
possessing the skills they are trying to achieve. The 
lecturer should be presenting himself for scrutiny. Using 
the course content as a vehicle, the lecturer demonstrates 
how he behaves and thinks relative to the subject matter. 
Thus, the lecturer is most effective when telling case 
histories and least effective when reading from the notes 
he hopes to convert into a textbook. 

It is important to retain some time in each course for a 
lecturer to present "war stories" about police work. As 
the lecturer is no longer viewed as responsible for presenting 
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the entire course content, he can select topics for his 
lectures that are most interesting, exciting, or illustrative. 

The atmosphere of the lectures can become more relaxed, 
and they can be successfully offered by persons previously 
considered to be poor showmen. Good lecturers have always 
been those individuals who measured their lectures by their 
personal performance. These are the lecturers who are 
willing to spend the time to tell a story that illustrates a 
point. All experienced police officers have anecdotes relating 
to police problems which derive from their own experience. 

Many can be good lecturers if they will abandon the attempt 
to be talking textbooks. 

A few hours of lecture were retained in the Penal Code and 
Arrest, Search and Seizure courses to permit the recruits 
to hear and see a trained person address the same problems 
they must learn to handle themselves. Additional lecture time 
was devoted to guest lecturers, who provided a non-police view 
of the criminal justice system. 

Both courses were also supplemented with role playing 
exercises. This excellent educational device is becoming 
increasingly widespread and should be of enormous benefit. 

The OPD has video taping equipment which is used to record, 
play back, and critique role playing incidents. Role playing 
is an excellent way to use the class time freed by the 
efficiency of the PI materials and is a good supplement to 
the courses on the Penal Code and on the Laws of Arrest, Search 
and Seizure. 

During the course on the Laws of Arrest, Search and Seizure, 
two of the training films produced by the Los Angeles District 
Attorney's Office were shown and discussed. This series, 
which includes over 20 films, is an excellent and inexpensive 
source of dramatized incidents for analysis by the students. 

The films were not designed to elicit active student responses, 
but that defect can be remedied by stopping the film frequently 
to allow time for students to reach their own evaluations of 
specific incidents. 

The PI texts on the law made available as a result of this 
program, free the instructor in a beginning peace officer training 
program from the need to lecture on all the specific points of 
course content. This reduces his preparation time. By 
replacing lecture and undirected home study with individualized, 
self-paced programmed instruction lessons, the student 
learns more easily and more quickly. The student time made 
available by these efficiencies should be devoted to enriching 
the course of study through more discussions, more guest 





lectures, and more role playing exercises or to providing 
more time for other courses in the curriculum. Though 
the PI materials could be used exclusively for homework 
assignments, instructors are urged to try the supervised 
study hall approach. It gives them time to tutor individual 
students without wasting the time of the remainder of the 
class and provides the students with a chance to talk over 
the material as they are learning it. If used in this way, 
the PI materials allow the instructor to lecture about 
the proper application of the information instead of on the 
infprmation itself. Thus, lectures are used to their best 
advantage to cover the vital and interesting aspect of 
how an experienced officer actually applies the law in 
specific cases . 

Each instructor should review any PI text that he will use 
to find those points in which his department's policies or his 
own opinions differ from those presented by the authors. 

These points should be handled during lecture periods. For 
example, there are differences in policy concerning the 
advis eabilitv of taking a drunk home rather than arresting 
him. Once the students learn the law from the PI text, they 
will readily understand local policies intended to provide 
police service meeting the needs of their community. Thus, 
it is hoped that a few specific comments by the instructor 
on policy and dis c.retionary points will easily correct any 
differences between the views in the PI texts and those of 
any agency in California. 

2.2.2 In-Service Training Applications 

The self-study format of PI texts makes them particularly 
flexible for in-service training, where manpower shortages 
make it difficult to free men from their assignments for 
training purposes. A whole range of possibilities exist, 
ranging from unmonitored use of the PI materials for homestudy 
to their use in conjunction with a regular training course. 

Within the Oakland Police Department, a pilot program of 
in-service training during roll call, that did not require 
. removing men from their regular assignments, was 

implemented, as described in the following departmental 
order. 
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OFFICE OF CHIEF OF POLICE 
OAKLAND POLICE DEPARTMENT 

SPECIAL ORDER NO. 1567 

TO: All members. Bureau pf Field DATE: 13 Apr 71 

Operations and Community Relations 
and Youth Division 



SUBJECT: Lineup Training in Criminal Law 

EXPIRATION DATE: 6 Jun 71 

I . General Information 

For the past eight months, the Department has been in- 
volved in a federally-funded grant project to develop 
programmed instruction training materials on Criminal 
Law. Considerable effort has been expended to prepare 
study guides and course workbooks by Lieutenant Joseph 
Colletti of Preventive Services Division, in conjunction 
with educational specialists working under contract to 
the Department . 

Seven programmed lessons and a test covering the entire 
Penal Code have been developed and are presently being 
used to streamline the teaching of this subject to inex- 
perienced officers undergoing recruit training. The 
next step in the development and testing of the program- 
med instruction method is use of the materials for in-ser- 
vice refresher training of experienced officers. 

II . Method of Conducting In-Service Training 



During a seven week period beginning 19 Apr 71 , . sergeants 
and patrolmen of the Bureau of Field Operations and the 
Youth Section will be given self-instruction lessons on 
the Penal Code, to be completed during regular line-up 
training periods. Completion of the lessons will provide 
both a review and updating of personal knowledge of the 
Penal Code and an evaluation of the effectiveness of the 
materials and of programmed instruction as a method of 
in-service training. 

* Approximately forty officers will be selected at random 
and tested on their knowledge of the Penal Code before and 
after the programmed instruction lessons are administered 
to sergeants and patrolmen of these organizational units 






in order to measure the value of the training program. 

Since the tests will be designed strictly as an eval- 
uative tool of the training material, individual scores 
will be held in confidence. 

Training Division Responsibilities 

A member of the Training Division will attend all line- 
ups on 19, 20 and 21 Apr 71 to distribute self-instruction 
workbooks containing the first programmed lesson to all 
sergeants and patrolmen of the aforementioned organizational 
units, and to explain procedures for completing the work- 
books and documenting time devoted to the project. The 
Training Division will provide organizational unit command- 
ers with copies of the Penal Code to be used in conjunction 
with the lessons and Labor Distribution Cards to document 
time spent by participating members. 

Responsibilities of Participating Members 

A. Completing Programmed Instruction Lessons 

Specific instructions for using the programmed materials 
will be included as a part of the lessons or provided 
when the lessons are distributed. Each of the seven 
lessons which will be provided are designed for self- 
study to enable participants to work at their own speed. 
Students will be able to move quickly over topics with 
which they are familiar and to spend more time on new 
laws or legal points which are not frequently encounter- 
ed on the job. The programmed lessons have been design- 
ed so that one lesson should be completed each week. 

Because of the variance of line-up training time avail- 
able for the project, members assigned to Patrol shall 
devote five one-quarter-hour line-up training sessions 
per week to programmed instruction. Members assigned 
to Traffic, Preventive Services and Youth Section shall 
. devote three one-half-hour sessions per week to program- 
med instruction. 

B . Labor Distribution Cards 

1 . Documentation of Time Spent 

■ ' // v 

Each member who has been issued a programmed work- 
book will be given a pre-printed labor distribution 
card in duplicate on which will be shown his name, 
city employee number and other data. Participating 
members will sign each card on the line above the 
words "Submitted By." The actual number of hours 



expended will be computed by the Training Div- 
ision . 

2 . Absences from Line-up 

If a member participating in the program is absent 
for any reason (e.g., O.T.A., illness, vacation) 
from line-up training sessions during the week for 
which the labor distribution card is provided, he 
shall sign both copies of that week's card on the 
line- above the words "Submitted By" and shall also 
indicate on the same line the number of line-up 
training sessions he actually attended that week. 

The actual number of hours expended will be computed 
by the Training Division. 

C. Responsibilities of Sergeants 

• Sergeants in attendance at line-ups shall distribute 
to each member the current week's programmed lesson 
and ensure that each member (including themselves) 
sign a labor distribution card in duplicate , to doc- 
ument time spent at line-up training during the prev- 
ious week. Bureau of Field Operations sergeants will 
deposit the signed cards in a box provided for that 
purpose . 

V. Organizational Unit Responsibilities 

Organizational unit commanders will be responsible for 
distributing subsequent lessons and collecting labor dis- 
tribution cards from members under their command. 

A. Bureau of Field Operations Responsibilities 

The Bureau of Field Operations commander shall ensure, 
on a weekly basis , that each sergeant and patrolman 
who was shown to be in an on-duty status by the daily 
watch detail signs a labor distribution card. The 
Bureau commander shall ensure that all signed time 
cards are forwarded to the Training Division within 
three days following completion of the preceeding 
week's lesson. 

B . Youth Section Responsibilities 

The Youth Section commanding officer shall ensure that 
all sergeants, patrolmen and policewomen under his com- 
mand sign the labor distribution cards at the conclus- 
ion of each week's lesson. The commander shall ensure 
that all signed time cards are forwarded to the Training 
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Division within three days following completion of the 
preceeding week's lesson. 



The materials may be retained after they have been .completed 
for future reference or study, with the exception of the Penal 
Codes, which shall be returned to the Training Division by or-' 
ganizational unit commanders. 

Questions regarding the program or critical comments after it 
has been completed should be directed to Lieutenant J. MbArthur 
Training Division. 



By order of 
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There was a wide variation in response to the pilot 
roll call training program using PI materials. Some 
watch commanders were energetic in seeing that the 
allotted 15 minutes were as free of interruptions as 
routine business would allow. Others were less co- 
operative with the program. Officers interested in 
study for promotional exams were eager students. 

Others thought that this training was a waste of their 
time. In this pilot program, there was no attempt to 
require the officers to complete the lessons or to 
concentrate on the task. The results of the pilot 
program were evaluated in two ways. 

Twenty officers were given a pre-test on the Penal Code, 
At the end of the training program, a second exam 
covering the same material was administered. The two 
exams are included in the appendix to this report. Only 
14 of the original 20 officers were available for the post- 
text. For this group of 14, the results were: 



This represents a gain in knowledge as measured by testing 
of 19%» No penalty for wrong answers was applied as a 
guessing factor. If that had been done, the improvement 
in test scores would have been greater. The pre-test 
average for the 20 officers taking that test was 7 3. 3%. 
Sixteen officers took the post-test, including two who had 
not taken the pre-test. The 16 officers had an average 
score of 84.6% on the post-test. This mixed grouping 
showed a test score improvement of 15.5%. 

A score improvement percentage is computed as: 



as computed with this formula were 4%, 5%, 6%, 8%, 8%, 

9%, 16%, 19%, 26%, 29%, 31%, 34%, and 39%. The average 
of these gains is 18%. It should be noted that every member 
of the group showed some improvement and that some showed 



Pre-test average 
Post- test average 



71 . 4 % 

85.0% 



100 X (Post-test score - Pre-test score) 
Pre-test score 



For the 14 officers who took both tests, their improvements 



great change. In fact, seven men receiving the lowest grades 
on the pre-test had an average improvement of 26%., Three of 
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the men in the lower half of the class on the pre-test 
moved into the upper half on the post- test. These results 
indicate that the PI materials were particularly helpful 
to those needing the most help. A claim which cannot 
usually be made by conventional instructional techniques 
but which is common for PI materials. This is most 
appropriate for vocational training, where the organization 
wishes to ensure at least a specified level of competence 
on the part of every employee. 

Since officers are best informed about laws involved in the 
cases they handle most frequently, these test scores do 
not correspond directly to the percentage of cases which 
an officer is prepared to handle properly. Still, it is felt 
that in a significant number of cases these officers will 
have a sounder grounding in the law as a starting point 
for proper police action. 

An interesting feature of the errors made on the post-test 
was that they were almost entirely on material covered by 
the last 3 lessons of the PI text. On the pre-test the errors 
were more uniform over material on all topics. One might 
suppose that this was the result of the men having completed 
only the first 4 lessons but not the last 3. Such an occurrence 
could correspond to either a loss of interest or to the fact 
that insufficient time had been provided for studying during 
the 7 weeks of the roll call training program. 

Depending upon the reason for the observed text results, 
it might be expected that the effectiveness of the program 
could be increased by either allowing more time than 7 weeks 
for the project or by providing some monitoring of progress 
as a stimulus for increased effort. Perhaps both would be 
useful. Merely, the administration and grading of pr*e- and 
post- tests would tend to motivate student efforts. 

The second measure of the in-service training program was 
an opinion poll conducted by the Department's Training 
Division. The opinion questionaire is included in the appendix 
of this report. The results of the poll are presented in the 
Training Commander's letter which appears in section 1.2.6 
of this report. 

The pilot program using PI materials for in-service training 
was conducted with the lessons of the text on the Penal Code. 
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Though specifically developed to meet the needs of 
recruits, 77% of the experienced officers polled stated 
that they thought the level of the material was satisfactory. 

The level of the text on arrest, search, and seizure will 
probably be even more acceptable to experienced officers 
than the Penal Code course, because it contains a large 
number of realistic practice cases and deals with an 
area of practical importance which has undergone significant 
changes in recent years. The Arrest , Search and Seizure 
text will be used in a forthcoming in-service training program 
by .the Oakland Police Department following the model of the 
pilot program on the Penal Code. The text on Dangerous 
Weapons Control Laws would also be suitable for such use. 

The course on Evidence is intended to provide an adequate 
orientation to the procedures of prosecution to enable 
the recruit to begin to perform as a peace officer. Once 
an officer has been involved in a trial, he may find this 
course too simple. The course contains a test on each 
chapter, which enables the student to determine whether 
he has mastered the contents of each chapter. Thus, an 
experienced officer could move quickly through the Evidence 
text without wasting time. But, of the four, the Evidence 
text has the least to offer the experienced officer. 

As a result of the experience gained from the pilot program 
of in-service training, it appears that PI materials can be 
used effectively even during 15 minute sessions under the 
less the ideal study environment of the roll call period. 
Experienced officers, more than recruits, are able to handle 
the materials almost entirely on their own and have established 
channels for getting answers to legal questions that they may 
have. It is suggested that officers receiving such in-service 
training be given a pre-test and a post-test, primarily as a 
mechanism for showing official concern for their improvement. 
If the pre-test papers are collected and the students are not 
forewarned, there is no particular reason that the same 
questions cannot be used on both the pre- and post- tests 
(though different exams were used in this study). 
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2. 3 Computer Assisted Instruction 



2. 3. 1 Presentation of Instruction by Computer 

In this project, the use of a computer as an automated tutor 
was investigated. Two modes of operation were tested. One 
is described in this section and one in section 2. 3.3, In 
the first, the programmed instruction materials originally 
developed in written form were "programmed" into the 
computer for presentation to the student at a TV style 
terminal such as is shown in Figures 1 and 2. The 
material appears in written form on the screen and the 
student answers by typing his response on the keyboard of 
the terminal. The student's response is automatically 
compared to "correct" response and he is informed by 
printed message whether his response was correct or 
incorrect. Depending upon his answer, the next occurance 
will be either to move on to next portion of the course, to 
go back to an earlier part of the course, to receive a 
remedial comment before one of the first two options, 
or to be given another chance to make the correct response. 

These choices are pre-determined by the educational 
programmer when he prepares the PI materials for 
insertion into the computer. 

In this project, the word "program" is used in two different 
ways. Instructional rra terials are "programmed" as 
described in earlier sections by developing a sequence of 
information presentation which requires active participation 
by the student. Such PI materials can be presented in the 
form of texts, films, or audiotapes. In the type of computer 
assisted instruction used in this project, the written PI 
materials are printed at a computer terminal each time a 
sutdent goes through the lesson. In order to control the 
operation of the computer, a computer program must be 
prepared by a computer programmer. The computer program 
consists of instructions which the computer executes in 
presenting the PI materials to the student. Both the computer 
program and the PI materials are stored in the computer 
memory. 

Instructions on the use of the computer program used in this 
project for computer presentation of PI materials is described 
in the appended set of CAI Documents. That program can 
present any PI materials on any subject. The computer program 
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Figure 1 Photograph of a computer time sharing terminal. 

The Bunker- Ramo terminal pictured is typical of 
the type of terminal which will be used for this 
program. 
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Figure 2 
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Photograph of a i police officer engaged in 
computer assisted instruction at a computer 
terminal. The terminal communicates with 
the computer by audible beeps transmitted 
by ordinary telephone. The computer displays 
written messages on the television screen 
(see Figure 1) and the student responds by 
typing on the electrified keyboard. Operation 
of the terminal is silent. The student is shown 
referring to the Penal Code as instructed by 
a message from the computer. 
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is separate and distinct from the PI materials. To 
prepare printed PI materials for CAI, it is necessary 
to follow the instructions just mentioned. This is done 
by an educational technologist who makes educational 
decisions about the sequence of steps to be followed by 
the computer. The third lesson of the Penal Code course 
(covering Homocide) was prepared for computer presentation. 
It is shown in the form ready for storage in the computer 
memory in the appended set of computer documents. Each 
line of the course is identified by a two letter code which 
indicates to the computer the nature of the text on that 
line, such as a question or an answer. Once the educational 
technologist gets the PI materials into this form, following 
the instructions mentioned above, the PI material is 
keypunched and fed into the computer for storage. 

Many of the responses required of the student in the PI 
materials are just short answer, such as "YES" or "TRUE" 
or a letter indicating the correct item on a multiple choice 
question. However, ,some of the educational objectives 
required could only be met by "constructed" responses, in 
which the student writes out his answer. Since the computer 
can only identify correct answers if theyexactly match the 
answer originally specified by the educational technologist, 
a constructed response of "malicious and willful" would be 
marked wrong if the correct answer were "willful and 
malicious". Similarly, the computer would treat "willful 
and malicious" as a wrong answer. 

The following innovation was used to circumvent this inherent 
limitation of CAI, in those cases which educationally called 
for constructed responses. The student was instructed to 
make his constructed response in writting on a piece of paper 
and then to indicate to the computer that he was finished 
answering by hitting the attention key on the terminal. At 
that point, the computer displayed the stored correct answer 
on the screen and asked the student if his answer were correct. 
Our experience with self-study PI materials indicates that 
students recognize that individualized study is not a test and 
can therefore be trusted to play fair in grading their own 
responses. In working with printed PI materials, the 
student grades all of his own responses. It therefore seemed 
reasonable to allow him to grade hii> own constructed 
responses in the CAI situation. After the student indicates 
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"YES" or "NO" to the question of whether his constructed 
response matches the stored correct answer, the computer 
proceeds by considering "YES" to be the correct answer 
to the original question and "NO" to be the wrong answer. 

The above approach for handling constructed responses in 
CAI was found to work well. It eliminates both the 
difficulty of requiring the student to type out a long answer 
and the problem that arises because the computer cannot 
interpret answers (but only compares them, letter by letter, 
with a previously stored correct answer). 

In the case of some concepts covered by the PI materials, 
the remedial steps to be taken if a student makes an 
incorrect response depend upon which wrong response is 
made. The computer program allows the educational 
programmer to specify different sequences of events for 
various, specific wrong answers. For example, if the 
correct answer for a multiple choice question is item "A", 
a different sequence of instruction can be specified for 
students who respond "B" from that specified for students 
who respond "C". While this elaboration is time consuming f 
and not generally worthwhile, it is very helpful on occasion 
to assure that the student who makes an error is getting 
pertinent remediation that reduces his confusion. 

The computer system used had the capability of writing on 
the terminal screen at the rates of 10 letters per second and 
30 letters per second. These rates correspond roughly to 
100 words per minute and 300 words per minute. At the 
slower speed, the time required to print the material on 
the screen, exceeds the time required by the student to read 
it. Originally, it was speculated that the slow writing speed 
might advantageously slow the student down, forcing him 
to concentrate on every word. However, it was found that 
the slow speed was distracting and irritating. Most students 
reread the material once it was on the screen and many 
paid no attention to the material until it was all written out. 
Thus, the time lag created by the slow writing speed was 
lost time, in addition to being an irritation. Therefore, 
after a few initial trials, all CAI work was done at the 
faster speed of 300 words per minute. The slow speed of 
100 words per minute is a standard speed of teletype 
networks . 
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The computer operation for CAI is done on a time- shared 
basis, with the computer handling many users simultaneously. 
During heavy computer usage hours in the late morning and 
late afternoon, a student may have to wait a few seconds or 
longer before the computer proceeds with the next step 
in his course. To avoid delays, most CAI trials were 
conducted in the evening. Of the seven students who were 
subjects in this CAI trial, all but one spent an average of 
between 5 and 6 minutes per page of material (based on the 
page length of the original PI text). The one exception was 
a trial at 100 words per minute, during a day when response 
delays were particularly bad. In that trial, the subject 
averaged 8 minutes per page. On the other hand, students 
averaged between 2. 5 and 4 minutes per page when working 
from the printed text. The difference in speed seems to be 
attributable to one or more of the following causes: 

1. Waiting for the computer to proceed to the next step. 

2. Waiting for the computer to write each segment of the 
course on the terminal screen. 

3. Time spent in typing an answer on the keyboard rather 
than simply checking it off or saying it to yourself. 

4. Possible distraction or confusion relating to the use 

of a computer and a terminal. The writing on the 
screen is light green on a dark greyish-green background, 
is entirely in capital letters, and is harder to read than 
a printed page. 

It was found that officers quickly learned to use the terminal 
and were able to teach other officers to do so. An introduction 
to CAI was included in the course, so that the student could 
become familiar with CAI before starting on his lessons. The 
user instruction cards that were prepared are shown in 
Figure 3. It took 15 to 20 minutes for one officer to show 
the next officer how to handle the terminal ard for that 
second officer to go through the introductory lesson on CAI. 

The opinions of the officers about the computer presentation 
of PI material varied greatly. Some were excited by the 
novelty of dealing with a computer and some recognized that 
working with the printed texts was faster. Generally, the 
response to the computer did not seem significantly better 
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INSTRUCTIONS FOR USING THE COMPUTER TERMIN AL 

VJHEN YOU ARE READY TO START 

1. Set the terminal switches to ON and REMOTE . Turn the coupler 
switch on (so the red light is on). 

2. Dial the computer (Oakland phone number is 835-3562) and sat the 
phone in the coupler. 

3. Watch for the flashing light to go on at the front of the coupler. 
If it does not go on, replace the phone and then repeat Step 2. 

4. When the computer asks for user code type PAAC101 and hit the 
return key. 

5. When the computer asks for the password type OPD and hit the 
return key-. 

6. At the next pause, type RUN TEACH/ RCC and hit the return key. 

?. If the computer asks DO YOU KNOW HOW I WORK? type YES (or 1) and 
hit the return key. (Mote: Sometimes this step is skipped.) 

8. 'When the computer asks for the course name type CHAP3/PAAC101 and 
hit the return key. 

/ (See Step 6 and 8) is on the same key as ? (last key - bottom row) 

1 is always number 1 - not the letter L as on the typewriter 

0 in OPD is letter 0; in PAAC101 is the number zero 



INSTRUCTIONS FOR USING THE COMPUTER TERMIN AL 

WHEN YOU ARE READY TO STOP 

1. If you want to stop at a point where the computer asks if you 
want to continue or to stop, follow the instructions on the 
screen and you will get to the end of the lesson. Then go to 
Step 3. 

2. If you want to stop in the middle of a lesson, type STOP as the 
answer to any question that requires a typed answe". Then go to 
Step 3. 

3. After getting to a stopping point (either from Step 1 or from 
Step 2 or from the ond of the course) type 3YE and hit the return 
key. Be sure to note page number where you stopped. 

4. After the computer has said goodbye and finished typing the time 
used, turn the terminal switch to OFF, turn the coupler switch off 
(the red light will go off), remove the phone from the coupler and 
place back on the phone base. 



Special Reminders 

Type 1 to answer true or yes — type 0 (zero) to answer false or no 
Type page number in following form. .PAGER, .to skip to a certain page 

Type STOP to get off the computer if you are r.ot at a regular stopping 

point. 

Figure 3. Two sides of the instruction card enabling the 

student to use the computer terminal on his own. 
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than to the printed PI materials, but this may not be 
significant in view of the small population tested with CAI. 

The material presented by CAI was the same as that in 
the printed PI text. The major advantage that might have 
been expected was that the novelty and liveliness of 
computer presentation would hold the student's attention 
better than printed materials. This was not found to be 
the case. All the students tested found working with the 
computer tiring and frequently requested breaks. It also 
seemed that the ability to score one's own responses that 
is inherent in the PI text is more appealing and provides 
better motivation than the automatic scoring of responses 
done by the computer. 

Another point, is that the student is able to look back in 
the text but is unable to "leaf through" the lesson at the 
computer terminal. Thus, the self-pacing and self-managing 
aspects of a PI text are somewhat reduced in CAI. The 
computer does not rush the student; he has an unlimited 
time to make each response. Still, because of the 
psychological pressure that the machine is "waiting" and 
the limitation of being unable to glance back at earlier 
pages, CAI diminishes the self-pacing advantages of PI. 

This seems to show up as decreased motivation on the 
part of the students, though again the sample population 
is too small for a really reliable conclusion on this point. 

2. 3. 2 The Costs of CAI 

The starting point for CAI is the preparation of PI materials, 
so the costs of CAI are always in addition to the costs of 
PI materials development. Then there is the cost for 
preparing the PI materials for computer presentation and, 
finally, the cost during actual student usage. 

In the current project, the development of the PI materials 
cost about $1, 000 per student hour of instruction; a figure 
which is at the low end of prevailing industry standards. 
Preparation of the PI material for computer presentation 
cost about $500 in addition. These are one time costs and 
could be amortized over a large number of students if 
CAI were used on a continuing basis. 




-40- 



43 



Operating costs for each student involved: 



1. Terminal rental at $140 per month 

2. Telephone charges at $0.80 per hour for local 
calls, and multiples of that rate if a toll call is 
required to reach the computer. 

3. Computer usage ^charges. On the commercial 
service used for this project, these were over 

. $16 per hour. Charges varied with the time of 
day and with the student's speed. These charges 
consisted of $1 1 per hour minimum charge, plus 
about $5 or more per hour based on the number of 
operations performed by the computer. An estimate 
is given below of projected costs if an agency 
acquired a computer solely for use in CAI. 

4. Other computer costs, consisting of storage charges 
if a commercial service is used or of operating 
personnel costs in the event an owned computer 

is used for CAI. The storage costs during this 
project were $240 per month for maintaining the 
course and program files in computer storage. 

Thus, the cost per student hour of CAI during the project 
were in excess of $20 per hour. This figure is not 
representative of the costs which would actually be 
incurred if a small scale, time- shared computer were 
leased or purchased and dedicated to the CAI application. 
There are a variety of ways of providing the hardware 
for CAI and of financing such hardware. The following 
figures are representative of the approach which currently 
seems most desireable and which can be used as a planning 
model for other ways of accomplishing the same purpose. 

The hardware required for a small CAI application might 
consist of the following items: 



Hardware 



Purchase Price 



4 student terminals 



$10,000 



Mini- computer with 12K 
memory and a control 
terminal 



23,000 



Disk d rive and 256K fixed 

disk $15,000 

Magnetic Tape Input/Output 

Capability 14,000 

Software for computer system 5, 000 

$67, 000 

Such a system could be leased on a one-year basis for 40% 
of the purchase price ($ and a one -year full maintenance 
and service contract would cost about $3, 000. Thus, the 
cost per year would be about $30, 000. If each of the 4 
terminals were in use for 500 hours per year, the cost 
per student hour would be about $15. This figure is 
comparable to that obtained by using a commercial 
time- sharing service. 

Economies can be achieved in a number of ways. Use of 
a longer term lease or of outright purchase of the 
hardware would reduce the yearly cost below 40% of the 
purchase price. If the equipment were purchased and used 
for 5 years, the cost would be about 30% per year including 
20% for initial purchase and 10% for the cost of money. 

A larger system with more terminals would provide lower 
cost per terminal. A system with 16 terminals would cost 
about $110, 000, lowering the purchase price from $16,750 
per terminal on the 4 terminal system to $6, 875 per 
terminal on the 16 terminal system. Finally, one might 
project more than 500 hours of usage per year for each 
terminal. In a large department or in a region containing 
a number of moderately big departments, just a few hours 
per man per hour could keep the terminals occupied for a 
major fraction of the 8, 700 hours in a year. For example, 
if a 16 terminal system were purchased for $110,000 and 
charged at 30% per year plus $6, 000 a service contract 
and if each terminal were used 1, 500 hours per year, CAI 
could be provided at a cost of $1. 63 per student hour. This 
would provide an availability of 139,000 terminal hours, of 
which it is assumed that 24,000 will be used. Thus, 8 hours 
of terminal time could be provided for each 3, 000 men at 
a cost of $1. 63 per student hour. If even larger systems N 
were used or if fuller utilization of the terminals were 
possible, the costs could be reduced even further. 
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It is estimated that large user groups, such as public 
school systems, can obtain computer assisted instruction 
at less than the $0. 80 per student hour that it generally 
costs for conventional public school education. In the 
police environment where 24 hour scheduling is imaginable, 
even lower projections are possible. These figures indicate 
that CAI costs can be made relatively minor compared to 
the salary paid the student in a vocational training setting, 
so long as there is a large enough group of students to make 
extensive use of the hardware and to offset the initial 
development costs. Thus, the question of whether to adopt 
CAI on a broad scale seems primarily answerable on the 
basis of its merits rather than its cost. From the current 
study, it appears that CAI offers no net advantage over 
printed PI materials for the presentation of instructional 
materials. The primary factors in this conclusion are 
the faster learning rates and better portability of the 
printed PI materials. 

Computerized Diagnostic Testing 

In the current project, the use of the automated grading 
and branching capabilities of CAI for diagnostic testing of 
students was investigated. A computer program was 
written to present test questions at the terminal, to 
compare student responses to correct answers stored 
in the computer memory, and to finally print out a score 
and the diagnostic messages which had been programmed 
for each test item that was missed. The computer program 
which was developed for this purpose is included in the 
appended volume of computer documents. An 18 item test 
on homicide was developed from the criterion test on 
lesson 3 of the Penal Code PI text. Each question was keyed 
to one or more pages of the PI study materials. Thus, for 
each question on the test which the student missed, the 
computer would write out a message indicating the pages 
of PI material that should be studied to correct the deficiency 
which caused the student to miss the test item. That test 
is also presented in the attached compilation of CAI documents. 

A number of recruits and experienced officers took the 
18 item test on homicide. Each student spending between 
15 and 20 minutes at the terminal to complete the test. All 
of the test items were of the "objective" type, so a guessing 
penalty equal to the percentage missed is probably appropriate 
and will be referred to in the following discussion of the results. 
The test was taken by 3 newly hired recruits without prior 
police training. They achieved raw scores of 50%, 60%, and 
65%. Applying a one for one guessing penalty, the adjusted 



scores would be 0%, 20%, and 30%. Three experienced 
officers were given the test and achieved raw scores of 
70%, 70%, and 75%. Again, applying the guessing penalty, 
these would be adjusted to 40%, 40%, and 50%. The 
recruit who scored 65% initially was retested after he 
had studied the first 3 lessons of the Penal Code text. 

At the second testing, his raw score was 30%. Applying 
the guessing penalty, his test grades improved from 30% 
to 80% (raw scores of 65% and 90%). Another recruit, who 
had not been pretested and whose only training was 
self-study from the Penal Code text while awaiting the 
start of the next recruit academy, scored 95% on the exam. 

The above test results are interesting, but do not bear on 
the major issue of whether use of the computer as a testing 
machine has advantages over conventional testing procedures. 
Still, it should be noted that a diagnostic test has questions 
covering the initial concepts of the subject matter and the 
final application of all the information in the topic covered. 
Thus, in a diagnostic test on homocide, any citizen of 
sufficient knowledge to qualify as a police recruit could be 
expected to have enough information to make better than 
random guesses on many questions. At the other end of 
the scale, several quite difficult questions were included to 
diagnose any terminal confusion the student might have with 
the subject. Thus, it might be expected that experienced 
officers without recent training or exposure to homocide 
cases would miss a fair percentage of these harder questions. 

The experience of using the computer as an aid in 
administering diagnostic tests seemed to indicate that the 
disadvantages of this approach outweigh the advantages for 
regular use in the police environment. The advantages are 
automatic grading and the novelty which might tend to impress 
officers with the sophistication and, by inference, the value 
of the training which they are to engage in after the diagnostic 
test is completed. Another potential advantage is that 
officers might be more likely to voluntarily test themselves 
if a computer terminal were continuously available to all. 
However, it is also possible that they will voluntarily make 
use of the criterion tests included in the PI texts, once these 
are in their hands. Of course, there is no evidence to 
believe that an officer who is attracted to a computer 
administered diagnostic test will carry out the study 
prescribed as a result of the test. 
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The students given the computerized diagnostic test 
covered the 18 test items in 15 to 20 minutes. The 
pre-test on the Penal Code was also administered to 
experienced officers and was of similar difficulty. It 
consisted of 37 items and occupied 30 to 40 minutes of 
student time. Though the population was quite small, it 
appears that computerized testing is about as fast as 
conventional testing. This is different from the rates 
for PI presentation, where CAI was slower than printed 
presentation. One possible explanation is that in 
computerized testing, the student was unable to go back 
to check his work. If students in fact spent about 25% 
of their time looking back over their answers on the 
printed test, the rates of completing test items would be 
comparable to .the rates of completing instructional 
frames, with the faster rates being achieved with printed 
materials. The desircability of eliminating review of a 
test is difficult to decide because students are generally 
conditioned to checking over their answers. Also, it is 
a tenuous argument to say that not looking back is more 
analogous to actual job performance, in view of the 
many artifacts in the test environment. 

All of the officers used as subjects for the computerized 
testing were attracted by the concept. One student 
commented that the fact that you could not go back to an 
earlier question to change an answer made the test a 
better measure of one's actual ability. This feature 
enables one to ask a sequence of questions of successive 
levels of difficulty to determine just where the student's 
ability lies, without having to avoid phrasing in one 
question that would prompt the correct answer to another 
question. 

The disadvantages of computer administered tests are 
mainly questions of cost and convenience. It is more 
difficult to prepare a test for computer presentation 
and one is essentially forced to use only "objective" type 
questions. Where training materials, once programmed 
into the computer, will require only occasional and minor 
updating, test questions must be continually changed 
if the test is to continue to be useful to the users. Secondly, 
the amount of usage that can be projected for computerized 
testing will be significantly less than for CAI. Therefore, 
the economies expected from full utilization of computer 
hardware will not be realizable where the computer is used 
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solely as a testing machine. 

The extra steps involved in putting a given examination 
on the computer and in arranging access to the terminal 
for the officers will in all but the largest departments, 
outweigh the effort needed to grade the exams and to 
prepare the study prescriptions manually. Computerized 
grading of answer sheets can be arranged economically, 
possibly through a local college or high school, eliminating 
the paperwork of grading even though the exam is presented 
in printed form. Also, the study prescription will more 
likely result in study if coupled with a little live exhortation 
and personalized advice on how to proceed. 

The results of this trial usage of computerized diagnostic 
testing suggest that it should probably be considered a 
workable frill that could be used to dress-up an in-service 
training program. A short test on a single topic, such 
as homicide, might require 15 to 20 minutes of terminal 
time. In all but the largest departments, terminals could 
not be kept busy just for testing. Thus, the costs would 
be $10 to $15 per terminal hour or $2.50 to $3.75 per 
student tested on a single topic. Since one might want to 
have each student tested on a number of topics, the 
computer costs would make computerized testing expensive 
without achieving sufficient offsetting benefits to justify 
the cost. Therefore, the computer presentation of 
examinations seems impractical for the police training 
environment. Computerized grading of exams, on the other 
hand, can generally be obtained very inexpensively and should 
be considered whenever the number of students being tested 
exceeds 50. 



3. Financial Note 



Financial reports on the project will be submitted separately 
by financial officers of the City of Oakland. The project was 
completed within the estimated budget. 
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Appendix 



None of the lest instruments included in this appendix arc 
recommended for* use. They were designed to parallel 
previously administered tests, so that comparisons could 
be made between students using the PI materials and those 
who had previously received conventional instruction. It 
is recommended that instructors prepare their own tests, using 
the criterion tests included in the PI texts for guidance. 
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